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EXECUTIVE SUMMARY 


In response to a severe recession that began in 2007, the U.S. Congress passed, and 
President Barack Obama signed into law, the American Recovery and Reinvestment Act of 2009 
(Pub. Law 1 1 1-5). At an estimated cost of $831 billion, this economic stimulus package sought 
to save and create jobs; provide temporary relief to those adversely affected by the recession; and 
invest in education, health, infrastructure, and renewable energy. States and school districts 
received $100 billion to secure teachers’ jobs and promote innovation in schools. This funding 
included $4.35 billion for Race to the Top (RTT), one of the Obama administration’s signature 
programs and one of the largest federal government investments in an education grant program. 
RTT awarded three rounds of grants to states that agreed to implement a range of education 
policies and practices designed to improve student outcomes. In particular, the program sought to 
improve student outcomes for high-need students, including English language learners (ELLs) 
(U.S. Department of Education 2010). 

Given the importance and size of the RTT grant program, the U.S. Department of Education 
(ED) commissioned this evaluation to address the following broad issues: 

• Whether states that received an RTT grant used the policies and practices promoted by RTT 

and how that compares to other states 

• Whether use of these policies and practices included a focus on ELLs and whether that focus 

on ELLs differed between RTT and other states 

• Whether receipt of an RTT grant was related to improvement in student outcomes 

No new funds were appropriated for RTT in the most recent federal budget (for fiscal year 
2016), so the future of the activities and policies that began under RTT is uncertain. However, 
findings on the first and second issues presented in this report remain useful to (a) policymakers 
who are interested in broader lessons learned from the program’s implementation and (b) 
educators who are considering how to proceed with education policy in light of the additional 
flexibility that the Every Student Succeeds Act of 2015 provides to states in terms of school 
turnaround, accountability, assessment, and educator evaluation systems. Findings on the first 
and second issues also provide useful context for interpreting the findings on the third issue. For 
example, if use of the policies and practices promoted by RTT is similar between states that 
received grants and states that did not, then it seems less likely that we would observe a 
relationship between RTT grant receipt and student outcomes. Findings on the third issue remain 
of interest to policymakers and educators who would like to better understand whether and how 
the large RTT investment might be related to changes in student outcomes. 

This is the final report on RTT for this evaluation. An earlier brief focused specifically on 
whether states adopted teacher evaluation policies promoted by RTT in spring 2012, and whether 
adoption of these policies varied across states that did and did not receive RTT grants (Hallgren 
et al. 2014). An earlier report covered all major policy and practice areas that RTT promoted, 
examining the extent to which states reported using these policies and practices in spring 2012, 
and whether usage differed across states that did and did not receive RTT grants (Dragoset et al. 
2015). This final report builds on the earlier brief and report by including an additional year of 
data (spring 2013) and by examining how receipt of RTT grants was related to student 
achievement over time. 
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Main findings 

We examined the extent to which RTT grantees and other states reported using policies and 
practices in six main areas: (1) improving state capacity to support school improvement efforts; 
(2) adopting standards and assessments that prepare students to succeed in college and the 
workplace; (3) building state data systems that measure student growth and inform instruction; 

(4) recruiting, developing, rewarding, and retaining effective teachers and principals; (5) turning 
around low-performing schools; and (6) encouraging conditions in which charter schools can 
succeed. We conducted two sets of comparisons: (1) we compared Round 1 and 2 RTT states 
(tenned early RTT states in this report) with states that did not receive RTT grants (termed non- 
RTT states or other states in this report) and (2) we compared Round 3 RTT states (tenned later 
RTT states in this report) with non-RTT states. We distinguish between Rounds 1 or 2 and 
Round 3 because of differences in the grants’ timing, funding levels, and scope for these groups 
of states. We found the following: 

• In four of the six areas examined, early RTT states reported using more policies and 
practices promoted by RTT than non-RTT states in spring 2013. The four areas were (1) 
adopting standards and assessments that prepare students to succeed in college and the 
workplace; (2) recruiting, developing, rewarding, and retaining effective teachers and 
principals; (3) turning around low-performing schools; and (4) encouraging conditions in 
which charter schools can succeed. 

• Later RTT grantees reported using more RTT-promoted policies and practices related 
to teacher and principal certification and evaluation than non-RTT states in spring 
2013. 

• Across all states, use of RTT-promoted policies and practices was most common for 
data systems and least common for teacher and principal certification and evaluation. 

In the data systems area, states reported using, on average, 76 percent of the RTT-promoted 
practices examined. In the teacher and principal certification and evaluation area, states 
reported using, on average, 26 percent of RTT-promoted practices examined. 

• Across the six areas, early RTT states reported using more English language learner 
(ELL)-focused policies and practices promoted by RTT than non-RTT states. No 

differences were found between later RTT states and non-RTT states. 

• Findings from spring 2012 and spring 2013 on states' use of RTT-promoted policies 
and practices were similar. For the most part, the spring 2013 findings presented in this 
report were the same as the spring 2012 findings presented in an earlier report from this 
evaluation (Dragoset et al. 2015). 

• There were no significant differences between RTT and other states in use of RTT- 
promoted practices over time. When we examined changes over time in states' use of 
RTT-promoted practices, we found no significant differences between RTT and other states. 

• The relationship between RTT and student outcomes was not clear. Trends in student 
outcomes could be interpreted as providing evidence of a positive effect of RTT, a negative 
effect of RTT, or no effect of RTT. 
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In sum, it is not clear whether the RTT grants influenced the policies and practices used by 
states or whether they improved student outcomes. RTT states differed from other states prior to 
receiving the grants, and other changes taking place at the same time as RTT reforms may also 
have affected student outcomes. Therefore, differences between RTT states and other states may 
be due to these other factors and not to RTT. Furthermore, readers should use caution when 
interpreting the results because the findings are based on self-reported use of policies and 
practices. 

Background 

The first three rounds of the RTT grant competition sought to encourage states to implement 
a range of policies and practices designed to affect all levels of the education system, with the 
ultimate goal of improving student outcomes. 1 The six topic areas described in the RTT 
application were (1) improving state capacity to support school improvement efforts; (2) 
adopting standards and assessments that prepare students to succeed in college and the 
workplace; (3) building state data systems that measure student growth and inform instruction; 
(4) recruiting, developing, rewarding, and retaining effective teachers and principals; (5) turning 
around low-perfonning schools; and (6) encouraging conditions in which charter schools can 
succeed. The RTT objectives in each topic area and the subtopics within each topic are detailed 
in Table ES.l. 

The RTT grants were awarded to states that both demonstrated a solid record of reform (for 
example, states that had improved student outcomes overall and by student subgroup and that 
had made progress in the past in the RTT reform areas) and presented strong plans in their RTT 
applications for furthering policies in these areas. Across the first three rounds of competition, 46 
states and the District of Columbia applied for RTT grants, and 19 applicants received grants. 2 
The Round 1 awards were made in March 2010, Round 2 awards in August 2010, and Round 3 
awards in December 2011. The 12 states selected in the first two rounds received awards ranging 
from $75 million to $700 million. 3 In the third round, which was open only to the nine finalists 
from the second round who had not yet received an RTT grant, awards were made to 7 states. 
Because these awards were smaller (ranging from $17 million to $43 million), ED required these 
states to focus on only a portion of the policies described in their Round 2 applications. 


1 Additional rounds of the RTT program focused on improving early learning and development programs for young 
children and supporting district-developed plans to improve student achievement. Those rounds were not a focus of 
this study. 

2 Alaska, North Dakota, Texas, and Vermont did not apply for RTT grants in any round. 

3 States’ award amounts varied based on their share of the nation’s school-age population and the budget they 
proposed in their application for accomplishing their specific plans. 
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Table ES.1. Objectives of the RTT grants, by topic area 


Improving state capacity to support school improvement efforts 


Articulating the state’s education reform agenda and local education agencies’ participation in it 
Building strong statewide capacity to implement, scale up, and sustain the proposed plans 
Demonstrating significant progress in raising achievement and closing gaps 


Adopting standards and assessments that prepare students to succeed in college and the workplace 


Developing and adopting common standards 

Developing and implementing common, high-quality assessments 

Supporting the transition to enhanced standards and high-quality assessments 


Building state data systems that measure student growth and inform instruction 


Fully implementing a statewide longitudinal data system 
Accessing state data and using it to inform key stakeholders 
Using data to improve instruction 


Recruiting, developing, rewarding, and retaining effective teachers and principals 


Providing high-quality pathways to certification for aspiring teachers and principals 
Improving teacher and principal effectiveness based on performance 
Ensuring equitable distribution of effective teachers and principals 
Improving the effectiveness of teacher and principal preparation programs 
Providing effective support to teachers and principals 


Turning around low-performing schools 


Authority to intervene in the lowest-achieving schools and local education agencies 
Turning around the lowest-achieving schools 


Encouraging conditions in which charter schools can succeed 


Eliminating restrictions on charter school creation and enrollment 
Refining authorization and monitoring processes 
Source: RTT application. 
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Research questions and study design 

This report was guided by the following research questions: 

• Are RTT states using the educational policies and practices promoted by RTT, and how does 
that use compare to the use of those policies and practices by other states? 

• Does use of these policies and practices include a focus on ELLs, and does that focus on 
ELLs differ between RTT and other states? Does use of these ELL-focused educational 
policies and practices differ based on characteristics that might affect the relevance of using 
these policies and practices, such as the percentage of ELL students in the state or the 
achievement gap between ELLs and other students? 

• Is receipt of an RTT grant related to improvement in student outcomes? 

The theory of action for RTT is that policy changes at the state level represent the first step 
in the process of changing the education system. Changes must occur at the state level before 
occurring at other levels, such as in districts, schools, and classrooms. For example, for a district 
to change its teacher evaluation system, a state might have to first make changes to its teacher 
evaluation policies and requirements. The changes made at all levels of the education system 
could then improve student achievement. The RTT study design is summarized in the box below. 

Prior to receiving a grant, RTT states differed somewhat from other states 

The RTT program sought to reward states that not only proposed strong reform plans but 
that also had a solid record of reform, so it is possible that RTT states differed from other states 
before they were awarded their grants. We compared these groups before receipt of the grants to 
better isolate changes that may have been due to the grant and found: 

• States that received an RTT grant were already using more of some RTT-promoted 
policies and practices before the grants were awarded. States that received an RTT grant 
received higher scores from grant application reviewers than other states that applied for 
RTT on pre-existing state policy conditions. In addition, RTT states reported using more of 
the policies and practices aligned with the RTT program at baseline than other states in the 
teacher and principal certification and evaluation area. Finally, later RTT states also reported 
using more policies and practices aligned with RTT at baseline in the area of school 
turnaround. 
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RTT study design 

Sample. The sample for the RTT evaluation included 50 states and the District of 
Columbia (DC). 

Data on educational policies and practices. To collect information on states’ use of the 
policies and practices promoted by RTT, we conducted structured telephone interviews with 
representatives from state education agencies. We interviewed 49 states and DC in spring 2012 (for 
a 98 percent response rate) and 50 states and DC in spring 2013 (for a 100 percent response rate). 
There were six interview modules (one for each RTT topic area). Respondents were state 
administrators most knowledgeable about each area. To facilitate comparisons between RTT and 
other states, we asked all states the same questions. During the spring 2012 interviews, we not only 
collected data on the current school year (2011-2012), we also collected data on three of the six 
reform areas in the year before the announcement and implementation of RTT (2007-2008). These 
three areas included (1) teacher and principal certification and evaluation, (2) school turnaround, and 
(3) charter schools. All data provided by states were self-reported and not independently verified by 
the research team. For these reasons and potential concerns about recall accuracy, readers should 
exercise caution when interpreting the data, particularly from 2007-2008. 

Data on student achievement. To examine the relationship between receipt of an RTT grant 
and student outcomes, we obtained publicly available data on state-level test scores from the 
National Assessment of Educational Progress (NAEP), a nationally representative assessment of 
U.S. students. 

Analysis of RTT implementation. To examine how use of policies and practices promoted by 
RTT compares between RTT states and non-RTT states, we conducted two types of comparisons: 

(1) early RTT states (Round 1 and 2) with non-RTT states and (2) later RTT states (Round 3) with 
non-RTT states. We distinguished between early and later RTT states because of differences in the 
grants’ timing, funding levels, and scope between these groups of RTT states. To summarize the 
large amount of data collected, we identified state interview questions that aligned with the policies 
and practices that RTT sought to affect. We determined how many policies and practices each state 
reported using and then calculated the average number of policies and practices for early RTT 
states, later RTT states, and non-RTT states. We then tested whether differences were statistically 
significant between each of the RTT groups and the non-RTT group in the average number of 
policies and practices reported. Because the goal of this analysis was to provide descriptive 
information about the actual levels of policies and practices used by RTT and non-RTT states in 
spring 2012 and spring 2013, the results were reported as raw (unadjusted) means; they were not 
regression-adjusted to account for any pre-existing differences between RTT and non-RTT states. 
Readers should exercise caution when interpreting findings from this analysis because any 
differences in states’ use of policies and practices were not necessarily caused by RTT. Differences 
could be due to other factors, such as pre-existing differences between RTT and non-RTT states or 
other changes that took place at the same time as RTT. 

Analysis of relationship between RTT and student outcomes. Because it was not possible 
to provide credible estimates of the effect of RTT on student outcomes, we conducted a descriptive 
analysis of student outcomes before and after the award of RTT grants. This analysis plotted the 
average outcomes for early RTT states, later RTT states, and non-RTT states in each year. As with 
the analysis of RTT implementation, readers should exercise caution when interpreting these findings 
because any differences in student outcomes were not necessarily caused by RTT. 
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In spring 2013, early RTT states reported using more policies and practices 
promoted by RTT than non-RTT states in four of the six areas examined 

In spring 2013, early RTT states, on average, reported using more policies and practices than 
non-RTT states in four areas (Figure ES.l): 4 

• Standards and assessments (72 percent of policies and practices reported by early RTT states 
compared to 46 percent for non-RTT states) 

• Teacher and principal certification and evaluation (37 percent of policies and practices 
reported by early RTT states compared to 20 percent for non-RTT states) 

• School turnaround (65 percent of policies and practices reported by early RTT states 
compared to 53 percent for non-RTT states) 

• Charter schools (6 1 percent of policies and practices reported by early RTT states compared 
to 4 1 percent for non-RTT states) 

The magnitude of these differences was between one and three practices for all but one area 
(teacher and principal certification and evaluation), where the difference was larger (about seven 
practices). Therefore, some of the differences may not be meaningful in tenns of how they 
affected education policy and outcomes in the states. 

The differences between RTT and other states observed in 2013 may not be due to the RTT 
grant but rather to differences in state education policy prior to grant award. In the teacher and 
principal certification and evaluation area, for example, early RTT states reported using more of 
the policies and practices promoted by RTT than non-RTT states in spring 2013, but also in the 
2007-2008 school year, before the RTT grants were awarded. This pre-existing difference 
suggests that RTT grants were awarded to states that were already using some of the promoted 
policies and practices rather than RTT awards causing the use of those policies and practices. 5 

In one of the six areas examined, teacher and principal certification and 
evaluation, later RTT states reported using more policies and practices 
promoted by RTT than non-RTT states in spring 2013 

In spring 2013, later RTT states reported using more policies and practices than non-RTT 
states in one of six areas promoted by RTT (the area of teacher and principal certification and 
evaluation) (Figure ES. I). 6 Later RTT states reported using 33 percent of policies and practices 
in this area, on average, compared to 20 percent for non-RTT states. The magnitude of this 


4 Chapter IV contains results for the individual policies and practices within each of these areas. 

5 For three areas (state capacity, standards and assessments, data systems), we do not have information on use of 
policies and practices prior to the awarding of RTT grants. Therefore, we cannot determine whether the observed 
differences between early RTT states and non-RTT states in these areas predated receipt of the RTT grant. 

6 Because Round 3 grants were narrower in scope, not all later RTT states focused on every area targeted by RTT. In 
addition, less time had elapsed between receipt of RTT awards and our spring 2013 interviews for the later RTT 
states than for the early RTT states. Because of these reasons and the smaller sample size (7 later RTT states, as 
opposed to 12 early RTT states), statistically significant differences are less likely to be found between later RTT 
states and non-RTT states than between early RTT states and non-RTT states. 
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Figure ES.1. Use of policies and practices promoted by RTT, by topic area 



(10 items) Assessments (8 items) and Principal Turnaround Schools 

(10 items) Certification (10 items) (4 items) 

and Evaluation 
(39 items) 


| Early RTT States (N=12) 


Later RTT States (N=7) 


Non-RTT States (N=32) 


Source: Interviews with state administrators in spring 2013. 

Note: The total number of policies and practices (shown in parentheses below each bar) differs by topic area. 

The number of policies and practices that we examined for each area was directly related to the amount 
of emphasis placed on each area in the RTT application criteria. This figure reads as follows (using the 
first bar on the left as an example): early RTT states reported using 71 percent of the policies and 
practices in the state capacity area. 

* Significantly different from non-RTT states at the 0.05 level, two-tailed test. 


difference was slightly more than five practices. As with the early RTT states, baseline data 
suggest a difference between these groups in this area existed before RTT grants were awarded. 


Across all states, use of policies and practices promoted by RTT was highest 
for data systems and lowest for teacher and principal certification and 
evaluation 

Use of RTT-promoted policies and practices was highest in the data systems area, in which 
states reported using, on average, 76 percent of the RTT-promoted practices examined (not 
shown). Use of RTT-promoted policies and practices was lowest in the teacher and principal 
certification and evaluation area. In that area, states reported using, on average, 26 percent of the 
RTT-promoted practices examined (not shown). When focusing on use of individual policies and 
practices, nearly ah states reported (1) having a state longitudinal data system (SLDS), and (2) 
identifying teacher shortage areas (not shown). In contrast, no states reported using the following 
policies and practices: (1) using results from evaluations of certification programs to provide 
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additional funds for, expand, or promote certification programs that were shown to be effective 
for teachers; (2) doing the same for certification programs that were shown to be effective for 
principals; and (3) publicly reporting results from evaluations of certification program 
effectiveness for principals. 

Across the six areas, early RTT states reported using more ELL-focused 
policies and practices promoted by RTT than non-RTT states in spring 2013, 
but there were no differences between later RTT and non-RTT states 

Early RTT states reported using more ELL-focused policies and practices promoted by RTT 
than non-RTT states. Out of the 12 ELL-focused policies and practices, early RTT states 
reported using 58 percent (7.0 practices) compared to 45 percent (5.4 practices) for non-RTT 
states. Later RTT states reported using 53 percent of these policies and practices (6.4 practices), 
which did not significantly differ from non-RTT states. 

Findings from spring 2012 and spring 2013 on states’ use of RTT-promoted 
policies and practices were similar 

Lor the most part, the spring 2013 findings presented in this report were the same as the 
spring 2012 findings presented in an earlier report from this evaluation (Dragoset et al. 2015). 
Two key differences were (1) early RTT states reported using more policies and practices than 
non-RTT states in five out of six areas in spring 2012 (with school turnaround being the 
exception), compared to four out of the six areas in spring 2013 (with state capacity and data 
systems being the exceptions); and (2) in spring 2012, there were no differences between RTT 
states and other states in use of ELL-focused policies and practices, but in spring 2013, early 
RTT states reported using more of these policies and practices than non-RTT states. 

There were no significant differences between RTT and other states in use of 
RTT-promoted policies and practices over time 

When we examined changes over time in states’ use of RTT-promoted policies and 
practices, we found no significant differences between RTT and other states. This finding was 
the same for each of the six topic areas examined and for the ELL-focused policies and practices. 

The relationship between RTT and student outcomes was not clear 

The trends in outcomes that we observed could be interpreted as providing evidence of a 
positive effect of RTT, a negative effect of RTT, or no effect of RTT. Therefore, it is not clear 
whether receipt of an RTT grant was related to changes in student outcomes. This uncertainty is 
partly due to the difficulty in separating the effect of RTT from overall trends in student 
outcomes and partly due to the limited amount of data available for the analysis. 

Conclusions 

It is not clear whether the RTT program influenced the use of policies and practices 
promoted by the program in RTT states. Although some differences between RTT and other 
states were observed, other factors could explain those differences. In particular, some 
differences in use of policies and practices promoted by RTT existed prior to states’ receipt of 
RTT grants. 
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Similarly, it is not clear whether RTT influenced student outcomes. This uncertainty exists 
because (a) other changes taking place at the same time as RTT reforms might also have affected 
student outcomes, and (b) the findings could be interpreted as providing evidence of RTT having 
a positive effect, negative effect, or no effect. 





